Over tting Explained

Over tting arises when model components are evaluated against the wrong reference distribution. Most modeling algorithms iteratively nd the best of several components and then test whether this component is good enough to add to the model. We show that for independently distributed random variables, the reference distribution for any one variable underestimates the reference distribution for the the highest-valued variable; thus variate values will appear signi cant when they are not, and model components will be added when they should not be added. We relate this problem to the well-known statistical theory of multiple comparisons or simultaneous inference. 1 Iterative Modeling Algorithms Iterative modeling algorithms (IMAs) generate a search space M of models by repeatedly selecting a model m( ) 2 M and adding a component ci from a list of components C = c1; c2; :::; cn to m( ), producing m( ; ci). For example, m( ) may be the regression equation ŷ = 3c3 + 1c1, and m( ; c5) is ŷ = 3c3 + 1c1 + 5c5. Generally, IMAs do not add every possible component to each model m( )|this would result in exhaustive search|but rather, they add the component that appears best according to some evaluation function xi = V(ci; m( );S). We call xi the score of component ci given model m( ) and a sample of data S. For example, V might compute information gain or classi cation accuracy for decision tree induction algorithms, F ratios for stepwise multiple regression algorithms, and so on. We may de ne a general IMA algorithm as follows: IMA: Initially,M contains the empty model m(). Now iterate: 1. Select a model m( ) 2 M 2. Remove components from C on logical grounds if necessary, producing C. For example, regression models shouldn't contain multiple occurrences of the same variable; whereas decision trees can in some circumstances. 3. Find the best component, cmax 2 C , the one with the highest value xmax = max(x1; x2; :::xn), where xi = V(ci; m( );S) 4. If xmax > TV , where TV is a possibly dynamic threshold value, then add cmax to m( ).